2 research outputs found
Stochastic Average Gradient : A Simple Empirical Investigation
Despite the recent growth of theoretical studies and empirical successes of
neural networks, gradient backpropagation is still the most widely used
algorithm for training such networks. On the one hand, we have deterministic or
full gradient (FG) approaches that have a cost proportional to the amount of
training data used but have a linear convergence rate, and on the other hand,
stochastic gradient (SG) methods that have a cost independent of the size of
the dataset, but have a less optimal convergence rate than the determinist
approaches. To combine the cost of the stochastic approach with the convergence
rate of the deterministic approach, a stochastic average gradient (SAG) has
been proposed. SAG is a method for optimizing the sum of a finite number of
smooth convex functions. Like SG methods, the SAG method's iteration cost is
independent of the number of terms in the sum. In this work, we propose to
compare SAG to some standard optimizers used in machine learning. SAG converges
faster than other optimizers on simple toy problems and performs better than
many other optimizers on simple machine learning problems. We also propose a
combination of SAG with the momentum algorithm and Adam. These combinations
allow empirically higher speed and obtain better performance than the other
methods, especially when the landscape of the function to optimize presents
obstacles or is ill-conditioned.Comment: 37 pages, 52 figures. arXiv admin note: substantial text overlap with
arXiv:1309.2388 by other author
Adaptive Discrete Communication Bottlenecks with Dynamic Vector Quantization for Heterogeneous Representational Coarseness
Vector Quantization (VQ) is a method for discretizing latent representations and has become a major part of the deep learning toolkit. It has been theoretically and empirically shown that discretization of representations leads to improved generalization, including in reinforcement learning where discretization can be used to bottleneck multi-agent communication to promote agent specialization and robustness. The discretization tightness of most VQ-based methods is defined by the number of discrete codes in the representation vector and the codebook size, which are fixed as hyperparameters. In this work, we propose learning to dynamically select discretization tightness conditioned on inputs, based on the hypothesis that data naturally contains variations in complexity that call for different levels of representational coarseness which is observed in many heterogeneous data sets. We show that dynamically varying tightness in communication bottlenecks can improve model performance on visual reasoning and reinforcement learning tasks with heterogeneity in representations